MULTIPLEXING APPARATUS AND METHOD, 
IMAGE OUTPUT APPARATUS AND METHOD, 
AND RECORDING MEDIUM 



CROSS-REFERENCE TO RELATED APPLICATIONS 

[0001] The present application claims priority from 

Japanese Application No. P2000-248493 filed on August 18, 2000, 
the disclosure of which is hereby incorporated by reference 
herein . 

BACKGROUND OF THE INVENTION 

[0002] The present invention relates to an apparatus and a 

method for multiplexing, an apparatus and a method for image 

output, and a recording medium, and particularly to an 
apparatus and a method for multiplexing, an apparatus and a 
method for image output, and a recording medium that enable a 
viewer to view a main part of a television broadcast program 
without an interruption by a commercial and the like. 

[0003] Since television broadcasting has become widespread, 

a viewer can enjoy sports programs and other broadcasts at 
home. However, in the case of commercial television broadcasts, 
a commercial break for promoting a product or service of an 
advertiser is inserted into a main part of the program at 
predetermined times. As a result, the viewer cannot enjoy the 
main part of the program, for example a sporting event, during 
the period the commercial break is being broadcast. 
[0004] Similarly, for example, if an important play, such 

as a goal in a soccer match, is broadcast on replay during the 
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main part of the program, the viewer cannot enjoy seeing the 
movement of players during the commercial break. 
SUMMARY OF THE INVENTION 

[0005] The present invention has been made in view of the 

above, and it is accordingly an object of the present 
invention to enable the viewer to view the main part of the 
program continuously. 

[0006] A multiplexing apparatus according to the present 

invention includes a first input unit operable to receive a 
first material; a second input unit operable to receive a 
second material; a first selecting unit operable to select one 
of the first material and the second material as material of a 
first stream; a second selecting unit operable to select 
another of the first material and the second material as 
material of a second stream; a control unit operable to 
control the first selecting unit so as to select the another 
material in place of the one material as the material of the 
first stream, and operable to control the second selecting 
unit so as to select the one material as the material of the 
second stream during a predetermined period of time; and a 
multiplexing unit operable to multiplex the first stream and 
the second stream. 

[0007] The multiplexing apparatus may further include a 

rate control unit operable to control a bit rate of the second 
stream to a value within a range that does not limit a bit 
rate of the first stream. 

[0008] The second material may be a commercial. 
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[0009] The second selecting unit may select a third 

material as the material of the second stream before selecting 
the one material, the third material being displayed 
simultaneously with the one material of the second stream. 
[0010] A multiplexing method according to the present 

invention includes inputting a first material and a second 
material; selecting one of the first and second materials as 
material of a first stream and selecting another of the first 
and second materials as material of a second stream; selecting 
the another material in place of the one material as the 
material of the first stream and selecting the one material as 
the material of the second stream during a predetermined 
period of time; and multiplexing the first stream and the 
second stream. 

[0011] A recording medium according to the present 

invention is recorded with a program for multiplexing a first 
stream and a second stream for output. The program includes 
inputting a first material and a second material; selecting 
one of the first and second materials as material of a first 
stream and selecting another of the first and second materials 
as material of a second stream; selecting the another material 
in place of the one material as the material of the first 
stream and selecting the one material as the material of the 
second stream during a predetermined period of time; and 
multiplexing the first stream and the second stream. 
[0012] An image output apparatus according to the present 

invention includes a selecting unit operable to select one 
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stream from among a first stream and a second stream; a 
control unit operable to control the selecting unit so as to 
select another stream from among the first stream and the 
second stream in place of the one stream during a 
predetermined period of time; and an output unit operable to 
output during one time the one stream selected by the 
selecting unit and during another time the another stream 
selected by the selecting unit. 

[0013] The control unit may control display such that an 

image included in the one stream is displayed as an image of a 
specified area, and an image included in the another stream is 
displayed as an image of an area smaller than the specified 
area . 

[0014] An image output method according to the present 

invention includes selecting one stream from among a first 
stream and a second stream; controlling the selecting step so 
as to select another stream from among the first stream and 
the second stream in place of the one stream during a 
predetermined period of time; and outputting the one stream 
during one time, and outputting the another stream during 
another time. 

[0015] A recording medium according to the present 

invention is recorded with a program for controlling an image 
output. The program includes selecting one stream from among a 
first stream and a second stream; controlling the selecting 
step so as to select another stream from among the first 
stream and the second stream in place of the one stream during 
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a predetermined period of time; and outputting the one stream 
during one time, and outputting the another stream during 
another time . 

[0016] The apparatus and the method for multiplexing and 

the program on a recording medium according to the present 
invention select the another material in place of the one 
material as the material of the first stream and select the 
one material as the material of the second stream during a 
predetermined period of time. 

[0017] The apparatus and the method for image output and 

the program on a recording medium according to the present 
invention select the another stream in place of the one stream 
during a predetermined period of time, and display the 
material included in the another stream as an image smaller 
than an image of the material included in the one stream. 
BRIEF DESCRIPTION OF THE DRAWINGS 

[0018] Fig. 1 is a block diagram showing a configuration 

of a multiplexing apparatus to which the present invention is 
applied; 

[0019] Fig. 2 is a block diagram showing a configuration 

of a hard disk recorder to which the present invention is 
applied; 

[0020] Fig. 3 is a diagram of assistance in explaining 

multiplexing principles of the multiplexing apparatus in Fig. 
1; 

[0021] Fig. 4 shows an example of multiplexing different 

from that of Fig. 2; 
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[0022] Fig. 5 illustrates a relation between two streams; 

[0023] Fig. 6 shows an example of a display of the two 

streams shown in Fig. 5; 

[0024] Fig. 7 is a flowchart of assistance in explaining 

the operation of the multiplexing apparatus in Fig. 1; and 
[0025] Fig. 8 is a flowchart of assistance in explaining 

the operation of the hard disk recorder in Fig. 4. 
DETAILED DESCRIPTION 

[0026] Fig. 1 shows a configuration of a multiplexing 

apparatus to which the present invention is applied. The 
multiplexing apparatus 1 has four coding units 11-1 to 11-4. 
The coding unit 11-1 encodes a video signal VI and an audio 
signal Al, which comprise a single stream (one program 
channel) . In order for the coding unit 11-1 to encode the 
video signal VI, the video signal VI is input from a video 
interface (I/F) 21-1 to a video compressing unit 23-1 via a 
switch 22-1. The video compressing unit 23-1 encodes 

(compresses) the video signal input via the switch 22-1 by an 
MPEG (Motion Picture Expert Group) 2 method, and then outputs 
the video signal to a secondary multiplexer 12. 

[0027] An audio interface 24-1 takes in the audio signal 

Al and then outputs the audio signal Al to an audio 
compressing unit 26-1 via a switch 25-1. The audio compressing 
unit 26-1 encodes (compresses) the audio signal input thereto, 
and then outputs the audio signal to the secondary multiplexer 
12 . 

[0028] Though not shown, the coding units 11-2 and 11-3 
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are formed in the same manner as the coding unit 11-1. The 
coding units 11-2 and 11-3 supply the secondary multiplexer 12 
with coded data of a second stream comprising a video signal 
V2 and an audio signal A2 and coded data of a third stream 
comprising a video signal V3 and an audio signal A3, 
respectively. 

[0029] Similarly to the coding unit 11-1, the coding unit 

11-4 has a video interface 21-4, a switch 22-4, and a video 
compressing unit 23-4 for processing a video signal V4 and an 
audio interface 24-4, a switch 25-4, and an audio compressing 
unit 26-4 for processing an audio signal A4 . 

[0030] The coding unit 11-4 encodes a video signal and an 

audio signal to be inserted into empty space in a transmission 
band. As the video signal to be inserted, the video signal VI 
from the switch 22-1 of the coding unit 11-1 as well as the 
video signal V4 are supplied to the switch 22-4. As the audio 
signal, the audio signal Al from the switch 25-1 of the coding 
unit 11-1 as well as the audio signal A4 are supplied to the 
switch 25-4. Though not shown, the video signals V2 and V3 or 
the audio signals A2 and A3 are also supplied from the coding 
units 11-2 and 11-3 to the switch 22-4 or the switch 25-4, 
respectively . 

[0031] The switch 22-4 selects either the video signal V4 

supplied from the video signal interface 21-4, the video 
signal VI supplied from the coding unit 11-1, the video signal 
V2 supplied from the coding unit 11-2, or the video signal V3 
supplied from the coding unit 11-3. The selected video signal 
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is supplied to the video compressing unit 23-4. The video 
compressing unit 23-4 compresses the video signal supplied 
from the switch 22-4. Similarly, the switch 25-4 selects 
either the audio signal A4 supplied from the audio interface 
24-4, the audio signal Al supplied from the coding unit 11-1, 
the audio signal A2 supplied from the coding unit 11-2, or the 
audio signal A3 supplied from the coding unit 11-3. The 
selected audio signal is supplied to the audio compressing 
unit 2 6-4. The audio compressing unit 2 6-4 compresses the 
audio signal supplied from the switch 25-4. The video signal 
compressed by the video compressing unit 23-4 and the audio 
signal compressed by the audio compressing unit 2 6-4 are 
supplied to a primary multiplexer 16. 

[0032] The primary multiplexer 16 is also supplied with 

data for broadcast taken in by an information interface 20. 
[0033] The secondary multiplexer 12 multiplexes the three 

streams supplied from the coding units 11-1 to 11-3, and then 
outputs the result to a primary CAS unit 14. The primary CAS 
unit 14 enciphers (scrambles) the stream supplied from the 
secondary multiplexer 12 using a predetermined key (a key 
corresponding to a key stored in an IC card 91 of a viewer 
(Fig. 2)), and then supplies the result to an MPEG-TS buffer 
15. The MPEG-TS buffer 15 outputs the data input thereto as an 
MPEG transport stream to a transmitter not shown in the figure. 
The transmitter modulates the multiplexed stream, and then 
broadcasts the multiplexed stream to each household via a 
satellite or the like not shown in the figure. 
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[0034] The primary multiplexer 16 multiplexes a fourth 

stream input from the coding unit 11-4 and the data stream 
input from the information interface 20. The result is output 
to storage media 17 formed by a hard disk, for example, to be 
temporarily stored therein. The secondary multiplexer 12 
multiplexes the three streams supplied from the coding units 
11-1 to 11-3 at specified bit rates based on the complexity or 
the like of the video signals and the audio signals of the 
streams. A CPU (Central Processing Unit) 19 monitors the bit 
rate allocated to each of the streams in the secondary 
multiplexer 12, and then calculates a remaining bit rate of 
the transmission band being used. On the basis of the 
remaining bit rate, the CPU 19 controls the bit rate of the 
fourth stream output from the storage media 17, and then 
supplies the stream to the secondary multiplexer 12. A 
secondary CAS unit 18 may encipher (scramble) the fourth 
stream output from the storage media 17 and then supply the 
stream to the secondary multiplexer 12, as required. 
[0035] Thus, the secondary multiplexer 12 multiplexes a 

total of four streams: the three streams supplied from the 
coding units 11-1 to 11-3 and the stream supplied from the 
secondary CAS unit 18. 

[0036] The CPU 19 further controls operation of the parts 

mentioned above on the basis of control data supplied from 
another apparatus not shown in the figure. 

[0037] When a magnetic disk 22, an optical disk 23, a 

magneto-optical disk 24, a semiconductor memory 25 or the like 
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is inserted into a drive 21, the drive 21 loads a program 
recorded on the magnetic disk 22, the .optical disk 23, the 
magneto-optical disk 24, the semiconductor memory 25 or the 
like so that the CPU 19 executes the program. 

[0038] A television signal multiplexed and broadcast by 

the multiplexing apparatus 1 is received and reproduced in 
each household by, for example, a hard disk recorder 51 as 
shown in Fig. 2. 

[0039] The hard disk recorder 51 in the example of Fig. 2 

receives the television broadcast signal by means of an 
external BS antenna, and then inputs an intermediate-frequency 
signal resulting from conversion by the BS antenna to a tuner 
81 of a front end unit 61. The tuner 81 converts the 
intermediate-frequency signal input thereto into a baseband 
signal, and then outputs the baseband signal to a digital 
demodulator circuit 82. The digital demodulator circuit 82 
demodulates the signal input thereto, and then outputs the 
result to an error correction circuit 83. The error correction 
circuit 83 corrects an error of the demodulated data input 
from the digital demodulator circuit 82, and then outputs the 
resulting data to a demultiplexer 62. 

[0040] The demultiplexer 62 selects one stream (program 

channel) from one transmission channel input thereto 

(including the four streams, that is, four program channels in 
this case) according to an instruction from the user, and then 
outputs the stream to a primary CAS unit 63. 

[0041] The primary CAS unit 63 deciphers (descrambles ) the 
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stream supplied from the demultiplexer 62 on the basis of a 
key stored in an IC card 91 inserted into an IC card interface 
64. When an instruction for recording has been issued by the 
user, a CPU 67 supplies the deciphered stream to a hard disk 
drive 65 for recording via a system bus 72. When an 
instruction for real-time reproduction has been issued, the 
output of the primary CAS unit 63 is supplied to a secondary 
CAS unit 69. The secondary CAS unit 69 deciphers (descrambles) 
the stream input thereto on the basis of a key that is 
prestored or obtained by accessing a specified server via a 
portable module 66, and then outputs the result to an MPEG 
decoder 70. 

[0042] The MPEG decoder 70 decodes the stream input 

thereto by an MPEG method. The MPEG decoder 7 0 then supplies a 
resulting video signal to an OSD (On Screen Display) circuit 
71 to multiplex an image based on specified OSD data. The 
result is thereafter output for display to a display such as a 
CRT or an LCD not shown in the figure. An audio signal output 
from the MPEG decoder 70 is supplied to a speaker or the like 
not shown in the figure. 

[0043] The CPU 67 controls operation of the above parts in 

response to an instruction from the user. When a magnetic disk 
74, an optical disk 75, a magneto-optical disk 76, a 
semiconductor memory 77 or the like is inserted into a drive 
73, the drive 73 downloads a program stored on the magnetic 
disk 74, the optical disk 75, the magneto-optical disk 76, the 
semiconductor memory 77 or the like to the hard disk drive 65 
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as required. Various data and programs are stored in a flash 
ROM 68 as required. 

[0044] The secondary multiplexer 12 in Fig. 1 multiplexes 

the streams in a manner as shown in Fig. 3, for example. In 
Fig. 3, the ordinate axis denotes a band of one transmission 
channel, and the abscissa axis denotes time. As shown in Fig. 
3, a television program PI forming the first stream, a 
television program P2 forming the second stream, and a 
television program P3 forming the third stream are coded at 
variable bit rates (VBR) by the coding units 11-1 to 11-3, 
respectively. The streams are multiplexed into one 

transmission channel having a predetermined bandwidth (for 
example 22 Mbps) by the secondary multiplexer 12. As a result, 
an empty space E is formed in the transmission channel. The 
fourth stream coded by the coding unit 11-4 is multiplexed 
into the empty space E. 

[0045] The capacity of the empty space E varies according 

to variations in the volume of the first to third streams. The 
empty space E is inherently not formed intentionally but 
created only as a consequence. This means that limitations on 
the bit rates of the first to third streams for securing a 
given space as the empty space E will not be imposed. On the 
other hand, the bit rate of the fourth stream to be inserted 
into the empty space E is controlled so as not to limit the 
bit rates of the first to third streams. 

[0046] To illustrate for comparison, the secondary 

multiplexer 12 does not multiplex the streams in a manner as 
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shown in Fig. 4. Specif ically, in the example of Fig. 4, each 
of the first to third streams formed by the television 
programs PI to P3 is coded within a range of a preset maximum 
bit rate. If the streams are multiplexed on the basis of this 
principle, each of the streams requires an empty space to 
spare. Therefore, it is practically difficult to transmit the 
fourth stream by using one empty space obtained by combining 
the empty spaces of the three streams. Thus, it may be said 
that in this case, one transmission band is divided into three 
constant bit rates (CBR) , and the streams are coded within 
their respective bands at variable bit rates . 

[0047] Fig. 5 schematically shows the relationship between 

the fourth stream (auxiliary stream) to be inserted into the 
empty space E and the first to third streams as the main 
streams . 

[0048] As shown in Fig. 5, in this example, the television 

program PI forming the first stream is, for example, a program 
of a soccer match being broadcast live, and therefore the 
television program Pi is basically formed by continuous sports 
scenes. However, a commercial is inserted during a specified 
time (in this case during a period from a time tl to a time 
t2). As a result, the first stream has a sports scene Pl-1 
broadcast until the time tl, a commercial CM1 broadcast during 
the period from the time tl to the time t2, and again a sports 
scene Pl-3 broadcast after the time t2 . Thus, a viewer of the 
first stream is not able to view a continuous sports scene 
during the period from the time tl to the time t2 . 
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[0049] According to the present invention, a sports scene 

Pl-2 whose starting point is continuous with the sports scene 
Pl-1 and whose ending point is continuous with the sports 
scene Pl-3 is broadcast as a television program PE of the 
fourth stream in the empty space E during the period from the 
time tl to the time t2 . During a period until the time tl, a 
commercial CM11-1, which is displayed simultaneously with the 
sports scene Pl-2 on the same screen, is inserted into the 
fourth stream. During a time period after the time t2, a 
commercial CM11-2 is disposed in the fourth stream. When a 
sports scene Pl-4 (not shown in the figure) succeeding the 
sports scene Pl-3 is disposed in the fourth stream, the 
commercial CM11-2 is displayed simultaneously with the sports 
scene Pl-4. 

[0050] Fig. 6 shows an example of such display. In this 

example, an image of the sports scene Pl-2 included in the 
fourth stream is displayed as an image of a slightly smaller 
area than that of a normal image. Commercial images CMll-1-1 
to CM11-1-9 comprising the commercial CM11-1 broadcast 
temporally before the sports scene Pl-2 are displayed as 
images smaller than the image of the sports scene Pl-2 on the 
right side and the lower side of the image of the sports scene 
Pl-2 . 

[0051] The multiplexing processing of the multiplexing 

apparatus 1 in Fig. 1 will next be described. The coding unit 
11-1 takes in the video signal VI by means of the video 
interface 21-1, and then supplies the video signal VI to the 
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video compressing unit 23-1 via the switch 22-1. The video 
compressing unit 23-1 encodes (compresses) the video signal VI. 
Similarly, the audio signal Al taken in by the audio interface 

24- 1 is supplied to the audio compressing unit 2 6-1 via the 
switch 25-1. The audio compressing unit 26-1 compresses the 
audio signal. Also, the coding unit 11-2 compresses the video 
signal V2 and the audio signal A2, and the coding unit 11-3 
compresses the video signal V3 and the audio signal A3. The 
first to third streams (television programs Pi to P3) output 
by the coding units 11-1 to 11-3 are supplied to the secondary 
multiplexer 12 to be multiplexed into one transmission channel, 
as shown in Fig. 3. 

[0052] On the other hand, the video interface 21-4 and the 

audio interface 24-4 of the coding unit 11-4 take in the video 
signal V4 and the audio signal A4, respectively, which form 
the fourth stream, and then supply the video signal V4 and the 
audio signal A4 to the video compressing unit 23-4 and the 
audio compressing unit 2 6-4 via the switch 22-4 and the switch 

25- 4, respectively. The video compressing unit 23-4 compresses 
the video signal input thereto. The audio compressing unit 2 6- 
4 similarly compresses the audio signal input thereto. The 
primary multiplexer 16 multiplexes the video data input from 
the video compressing unit 23-4, the audio data input from the 
audio compressing unit 2 6-4, and the data for broadcast input 
via the information interface 20 under control of the CPU 19. 
The result is supplied to the storage media 17 to be stored 
therein . 
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[0053] The CPU 19 effects control under control of an 

external apparatus so that the fourth stream is inserted into 
the empty space E as required. 

[0054] Thus, the CPU 19 monitors the bit rates of the 

first to third streams output by the secondary multiplexer 12, 
and thereby calculates the capacity of the empty space E. Then, 
the CPU 19 corrects the bit rate of the fourth stream so that 
the fourth stream may be inserted into the empty space E. 
[0055] The data of the fourth stream having the adjusted 

bit rate is read from the storage media 17, enciphered 
(scrambled) by the secondary CAS unit 18 as required, and then 
supplied to the secondary multiplexer 12. The secondary 
multiplexer 12 multiplexes the first to third streams supplied 
from the coding units 11-1 to 11-3 and the fourth stream 
supplied from the secondary CAS unit 18, and then outputs the 
multiplexed stream to the primary CAS unit 14 . 

[0056] The primary CAS unit 14 enciphers (scrambles) the 

whole of the multiplexed stream supplied from the secondary 
multiplexer 12, and then supplies the resulting multiplexed 
stream to the MPEG-TS buffer 15. The MPEG-TS buffer 15 
converts the multiplexed stream supplied from the primary CAS 
unit 14 into an MPEG transport stream format, and then outputs 
the result to a transmission apparatus not shown in the figure. 

[0057] When the commercial CMl is inserted into the first 

stream formed by the television program Pi at the time tl 
after the sports scene Pl-1, and the sports scene PI- 3 is 
reinserted into the first stream at the time t2 as shown in 
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Fig. 5, for example, the switch 22-1 of the coding unit 11-1 
switches data of the sports scene Pl-1, which has been input 
as the video signal VI, to data of the commercial CM1 input as 
a video signal Vll at the data input time tl- Then, at the 
time t2, the switch 22-1 switches from the commercial CMl, 
which has been selected as the video signal Vll, to the sports 
scene Pl-3 supplied as the video signal VI via the video 
interface 21-1. Thus, the first stream is switched from the 
sports scene Pl-1 to the commercial CMl at the time tl, and is 
further switched from the commercial CMl to the sports scene 
Pl-3 at the time t2. Switching between the audio signals is 
also performed in synchronization with the video signals. 
[0058] On the other hand, the switch 22-4 in the coding 

unit 11-4 selects the commercial CMl 1-1 supplied as the video 
signal V4 until the time tl, and then supplies the commercial 
CM11-1 to the video compressing unit 23-4. At the time tl, the 
switch 22-4 selects the sports scene Pl-2 supplied as the 
video signal VI from the switch 22-1, and then supplies the 
sports scene Pl-2 to the video compressing unit 23-4. At the 
time t2, the switch 22-4 selects the commercial CM11-2 
supplied as the video signal V4 from the video interface 21-4 
in place of the sports scene Pl-2 that has been supplied from 
the switch 22-1, and then outputs the commercial CM11-2 to the 
video compressing unit 23-4. 

[0059] Similarly to the switch 22-4, at the time tl, the 

switch 25-4 selects the audio signal Al corresponding to the 
sports scene Pl-2 supplied from the switch 25-1 in place of 
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the audio signal A4, and then supplies the audio signal Al to 
the audio compressing unit 2 6-4. At the time t2, the switch 
25-4 selects the audio signal A4 supplied via the audio 
interface 24-4 instead of selecting the audio signal Al that 
has been supplied from the switch 25-1, and then supplies the 
audio signal A4 to the audio compressing unit 26-4. 
[0060] Thus, the television program PE of the empty space 

E is switched from the commercial CMll-1 to the sports scene 
Pl-2 at the time tl, and is switched from the sports scene Pl- 
2 to the commercial CM11-2 at the time t2. 

[0061] Since the commercial CMll-1 and the commercial 

CM11-2 do not necessarily need to be displayed in real time, 
the lowering of their transfer bit rates does not present much 
of a practical problem when the commercials are temporarily 
recorded and then reproduced on the hard disk recorder 51 side. 
On the other hand, since the sports scene Pl-2 is continuous 
with the sports scene Pl-1 and the sports scene Pl-3, the 
sports scene Pl-2 needs to be displayed in real time. When the 
capacity of the empty space E is not sufficient to transmit 
data of a normal-size screen (full screen), reduction of image 
size (image frame) can lower the volume of the sports scene 
Pl-2 (an image of a full screen of M x N pixels is converted 
into an image of m x n pixels (m < M, n < N) ) . Thus, although 
the image becomes smaller, its real-time display is ensured. 
[0062] The hard disk recorder 51 can accommodate delay on 

the time axis by temporarily recording the received data on 
the hard disk drive 65 and then reproducing the data. A 
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receiving apparatus without such a built-in recording unit 
cannot make such adjustment on the time axis; therefore, the 
image frame is made smaller, and thereby its real-time display 
can be ensured. 

[0063] The operation described above is illustrated in the 

flowchart of Fig. 7. 

[0064] Specifically, at a first step SI, the CPU 19 

controls the coding unit 11-4 to multiplex the data of the 
commercial CM11-1 as data of the empty space E. At a step S2, 
the CPU 19 determines whether the entire stream of the 
commercial CM11-1 has been transmitted (multiplexed) . When the 
entire stream has not been transmitted, the process returns to 
the step SI to repeat the process at the step SI, 
[0065] When the CPU 19 determines at the step S2 that the 

entire stream of the commercial CM11-1 has been transmitted, 
the process proceeds to a step S3, where the CPU 19 transmits 
stream schedule data indicating stream switching at the time 
tl as part of the data for data broadcast. 

[0066] At the next step S4, the CPU 19 waits until the 

time for transmitting (multiplexing) the commercial CM1 
arrives. When the transmission time has arrived, the process 
proceeds to a step S5, where the CPU 19 calculates the 
capacity of the empty space E in the transmission channel on 
the basis of the spaces used by the first to third streams in 
the secondary multiplexer 12. 

[0067] At a step S6, the CPU 19 determines whether the 

capacity of the empty space E calculated at the step S5 
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provides a sufficient band to transmit the sports scene Pl-2. 
When the band is not sufficient (when the band is lacking) , 
the process proceeds to a step S7, where the CPU 19 performs 
processing to reduce the screen size of the sports scene Pl-2 
stored in the storage media 17. More specifically, the CPU 19 
converts an image of a full screen of M X N pixels into an 
image of m x n pixels. At a step S8, the CPU 19 compresses the 
image data stored in the storage media 17 as required. At a 
step S9, the CPU 19 multiplexes data specifying the screen 
size set at the step S7 and data specifying the arrangement of 
the commercials CMll-1-1 to 11-1-9 around the periphery of the 
image (sports scene Pl-2) of the reduced screen size as shown 
in Fig. 6. 

[0068] When the CPU 19 determines at the step S6 that the 

capacity of the empty space E is sufficient, the processes at 
the steps S7 to S9 are skipped. Specifically, in this case, 
the commercials CMll-1-1 to 11-1-9 shown in Fig. 6 are not 
displayed, and the image of the sports scene Pl-2 is displayed 
in full screen size. 

[0069] At the next step S10, the CPU 19 outputs data 

instructing a receiving apparatus (the hard disk recorder 51 
in this case) receiving and reproducing the stream of the 
television program Pi to switch to the stream of the empty 
space E at the time tl. 

[0070] At the next step Sll, the CPU 19 waits until the 

time for ending the commercial CM1 (time t2) arrives. When 
that time has arrived, the process proceeds to a step S12, 
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where the CPU 19 outputs data for switching the screen size 
back to the original full screen size. Also, at a step S13, 
the CPU 19 outputs an instruction to switch from the stream of 
the empty space E to the stream of the television program PI. 
[0071] Operation of the hard disk recorder 51 in Fig. 2 

will next be described. The tuner 81 converts the 

intermediate-frequency signal supplied from the BS antenna 
into a baseband signal, and then supplies the baseband signal 
to the digital demodulator circuit 82. The digital demodulator 
circuit 82 demodulates the baseband signal, and then supplies 
the result to the error correction circuit 83. The error 
correction circuit 83 corrects an error of the signal input 
thereto, and then outputs the result to the demultiplexer 62. 
[0072] The demultiplexer 62 is controlled by the CPU 67 

according to an instruction from the user to select the stream 
of a program channel included in the multiplexed streams of 
the transmission channel input thereto, and then outputs the 
stream to the primary CAS unit 63. When the stream input to 
the primary CAS unit 63 is enciphered, the primary CAS unit 63 
deciphers (descrambles ) the stream using a key recorded on the 
IC card 91. When an instruction for recording the deciphered 
stream has been issued by the user, the stream is supplied to 
the hard disk drive 65 for recording under control of the CPU 
67 . 

[0073] When an instruction for reproducing the stream has 

been issued by the user, the CPU 67 controls the hard disk 
drive 65 to reproduce the stream recorded on the hard disk 
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drive 65. The stream is input to the secondary CAS unit 69, 
and when the stream is enciphered, the secondary CAS unit 69 
deciphers the stream using a built-in key. When there is no 
built-in key, the CPU 67 controls the portable telephone 
module 66 to access a specified center via a telephone line. A 
key obtained from the specified center via the telephone line 
is supplied via the system bus 72 to the secondary CAS unit 69 
to be stored therein. 

[0074] When an instruction for real-time reproduction has 

been issued by the user, the stream output from the primary 
CAS unit 63 is directly supplied via the system bus 72 to the 
secondary CAS unit 69 to be deciphered. 

[0075] The stream deciphered by the secondary CAS unit 69 

is supplied to the MPEG decoder 7 0 to be decoded. A video 
signal resulting from the decoding of the stream is supplied 
to the OSD circuit 71. The OSD circuit 71 superimposes an OSD 
signal upon the video signal as required. The result is 
thereafter output to a display for display. An audio signal 
output from the MPEG decoder 70 is supplied to a speaker, 
[0076] The above process will be described in further 

detail with reference to the flowchart of Fig. 8. 
[0077] At a first step S21, the CPU 67 receives the data 

of the empty space E, and then supplies the data to the hard 
disk drive 65 for recording. Thus, the commercial CMll-1 in 
Fig. 5 is recorded on the hard disk drive 65. At a step S22, 
the CPU 67 determines whether the entire stream of the 
commercial CMll-1 has been received. When the entire stream 
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has not been received yet, the process returns to the step S21 
to repeat the process at the step S21. 

[0078] When the CPU 67 determines at the step S22 that the 

entire data of the commercial CM11-1 has been received, the 
process proceeds to a step S23, where the CPU 67 receives the 
stream schedule data transmitted at the step S3 in Fig. 7. 
According to the stream schedule data, the CPU 67 waits until 
the time for starting demodulation of the commercial CMl (time 
tl) arrives at a step S24. When the time has arrived, the 
process proceeds to a step S25. When the hard disk recorder 51 
receives the commercial CMl while receiving the television 
program Pi, the CPU 67 determines whether a subscription to 
receive and display the sports scene Pl-2 of the empty space E 
in place of the commercial CMl has been taken out in advance 
or not. When the subscription has been taken out, its 
information is prestored in the flash ROM 68. Thus, the CPU 67 
performs the determination process at the step S25 on the 
basis of contents stored in the flash ROM 68. 

[0079] When it is determined at the step S25 that a 

subscription to skip the commercial and view the continuous 
sports scene has been taken out, the process proceeds to a 
step S2 6, where the CPU 67 receives the data specifying the 
screen size transmitted at the step S9 in Fig. 7. On the basis 
of the data, at a step S27, the CPU 67 determines whether a 
reduction of the screen size has been specified. When a 
reduction of the screen size has been specified, the process 
proceeds to a step S28, where the CPU 67 reduces the screen 
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size, and also controls the OSD circuit 71 so that the images 
of the commercials CMll-1-1 to 11-1-9 are attached to the 
periphery of the image of the reduced screen size. 
[0080] When it is determined at the step S27 that a 

reduction of the screen size has not been specified, the 
process at the step S28 is skipped. 

[0081] At a step S29, the CPU 67 performs processing to 

switch from the stream of the television program Pi to the 
stream of the empty space E. Thus, the sports scene Pl-2 is 
displayed following the sports scene Pl-1. When the screen 
size of the sports scene Pl-2 is small, the commercials CM11- 
1-1 to 11-1-9 are displayed around the periphery of the sports 
scene Pl-2. 

[0082] At a step S30, the CPU 67 waits until the time for 

ending the commercial CM1 (that is, the time t2) arrives. When 
the time has arrived, the process proceeds to a step S31, 
where the CPU 67 receives a signal respecifying the screen 
size set at the step S12 on the transmitting side. At a step 
S32, the CPU 67 changes the screen size to the original screen 
size corresponding to the specification. At a step S33, the 
CPU 67 switches from the stream of the empty space E to the 
stream of the television program PI. Thus, the sports scene 
Pl-3 is displayed. 

[0083] When it is determined at the step S25 that the 

viewer of the hard disk recorder 51 has not taken out a 
subscription to skip the commercial, the process from the step 
S2 6 to the step S33 is skipped. More specifically, in this 
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case, the stream of the television program Pi is reproduced 
without the above processing; therefore, the commercial CM1 is 
displayed following the sports scene Pl-1, and then the sports 
scene Pl-3 is displayed. 

[0084] The series of process steps described above may be 

carried out not only by hardware but also by software. When 
the series of process steps is to be carried out by software, 
a program is installed from a network or a recording medium 
onto a computer where programs forming the software are 
incorporated in dedicated hardware, or a general-purpose 
personal computer that can perform various functions by 
installing various programs thereon, for example. 
[0085] Examples of the recording medium include not only 

program-recorded packaged media distributed to users to 
provide the program separately from the apparatus proper, such 
as the magnetic disks 22 and 74 (including a floppy disk) , the 
optical disks 23 and 75 (including CD-ROM {Compact Disk-Read 
Only Memory) and DVD (Digital Versatile Disk) ) , the magneto- 
optical disks 24 and 76 (including MD (Mini-Disk) ) , or the 
semiconductor memories 25 and 77, as shown in Fig. 1 or Fig. 2, 
but also a ROM and the hard disk drive 65 storing the* program 
and supplied to a user in a state of being preincorporated in 
the apparatus proper, 

[0086] It is to be noted that in the present specification, 

the steps describing the program recorded on a recording 
medium include not only process steps carried out in time 
series in the described order, but also process steps carried 
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out in parallel or individually and not necessarily in time 
series . 

[0087] In the present specif ication, a system refers to an 

apparatus as a whole formed by a plurality of units. 
[0088] As described above, the apparatus and the method 

for multiplexing and the program on a recording medium 
according to the present invention select a second material in 
place of a first material as a material of a first stream and 
select the first material as a material of a second stream 
during a specified period of time. Therefore, it is possible 
to allow a viewer to continuously view desired streams as 
required. 

[0089] The apparatus and the method for image output and 

the program on a recording medium according to the present 
invention select a second stream in place of a first stream 
during a specified period of time such that an image of a 
first material included in the second stream is displayed as 
an image of a smaller area than that of an image of the first 
material included in the first stream. Therefore, it is 
possible to display the image of the first material in real 
time . 

[0090] Although the invention herein has been described 

with reference to particular embodiments, it is to be 
understood that these embodiments are merely illustrative of 
the principles and applications of the present invention. It 
is therefore to be understood that numerous modifications may 
be made to the illustrative embodiments and that other 
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arrangements may be devised without departing from the spirit 
and scope of the present invention as defined by the appended 
claims . 
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